Skip to content

Interactive Learning Menu with AI Assistance #5307

Open
stutijain2006 wants to merge 5 commits intosugarlabs:masterfrom
stutijain2006:Interactive-Menu
Open

Interactive Learning Menu with AI Assistance #5307
stutijain2006 wants to merge 5 commits intosugarlabs:masterfrom
stutijain2006:Interactive-Menu

Conversation

@stutijain2006
Copy link
Contributor

This PR improves the Learning Guide experience in Music Blocks by adding context-aware AI help and making the guide more beginner-friendly and stable.

What's added in this PR-

  1. Interactive Learning Guide
  • Step-by-step guide that highlights palettes, buttons, and blocks
  • Prevents moving forward until the step is actually completed
  • Clear visual feedback for progress (completed / pending)
  1. AI Help Inside the Guide
  • Each step now has a “Need help?” AI section
  • Users can ask questions related to the current step
  • AI response depends on:
    1. the current step
    2. whether the step is completed or not
  • Explanations are simple, beginner-friendly, and practical
  1. Toolbar Access
  • Learning Guide is accessible from the toolbar
  • Users can start or close the guide anytime

Problems this PR solves-

  • Absence of an interactive onboarding menu for new users
  • Non-music learners being unfamiliar with musical concepts like tones, instruments, and palettes
  • First-time users struggling to understand how different palettes and blocks work together
  • The complexity of Music Blocks feeling overwhelming without step-by-step guidance
  • No built-in way to ask questions while learning, forcing users to rely on external resources

Setup of the AI Model-
Add a .env file in the project root:
OPENROUTER_KEY=your_api_key_here

Future Improvements-

  • Add more guide steps covering advanced palettes and features

A short demo video is attached to showcase the Learning Guide flow and AI interaction.

Screen.Recording.2026-01-24.140446.mp4

@github-actions
Copy link
Contributor

✅ All Jest tests passed! This PR is ready to merge.

@7se7en72025
Copy link
Contributor

Damn this is Crazy!!

@omsuneri
Copy link
Member

omsuneri commented Jan 24, 2026

@stutijain2006 this is similar to what we did last summer in the ai debugger may be you can think of it as a part of additional feature in the ai debugger project right now i dont think so we need this but if you want to do anymore such enhancements i request you to once review the ai debugger repo too and if you have any idea on this you can discuss with us on matrix !!

@stutijain2006
Copy link
Contributor Author

@stutijain2006 this is similar to what we did last summer in the ai debugger may be you can think of it as a part of additional feature in the ai debugger project right now i dont think so we need this but if you want to do anymore such enhancements i request you to once review the ai debugger repo too and if you have any idea on this you can discuss with us on matrix !!

Actually Om, I already went through that feature, but the major change I did is I am introducing an interactive menu which will particularly help the beginners to guide through the whole music block basic features and the purpose of integration of AI in the menu is to get the doubts solved at that moment only which will come into the mind while exploring the menu. The debugger function solves the ai doubt on the basis of the current workspace but motive behind integrating ai in menu is to get the doubts cleared which are coming on the way.
Also, it is very difficult for beginners to redirect to debugger in the widget section and even that was not working due to some ai credit or whatever.
So, the purpose served by ai model in interactive menu is completely different from ai debugger.

@github-actions
Copy link
Contributor

✅ All Jest tests passed! This PR is ready to merge.

@haroon10725
Copy link
Contributor

I think this feature is not in priority, as we have limited AWS credits.

@walterbender What do you think? If yes then we can close the PR.

@stutijain2006
Copy link
Contributor Author

I think this feature is not in priority, as we have limited AWS credits.

@walterbender What do you think? If yes then we can close the PR.

But instead of integrating AI in the interactive menu can't we integrate just the interactive menu if we have limited AWS credit and ai integration in this feature is not considered to be that important?

@stutijain2006
Copy link
Contributor Author

I think this feature is not in priority, as we have limited AWS credits.
@walterbender What do you think? If yes then we can close the PR.

But instead of integrating AI in the interactive menu can't we integrate just the interactive menu if we have limited AWS credit and ai integration in this feature is not considered to be that important?

Because we discussed the possibility of interactive menu to guide the beginners in the meet with @walterbender, and they found this idea to be good.

@walterbender
Copy link
Member

I don't think we need the AI integration here. But an interactive tutorial is not a terrible idea.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 3, 2026

✅ All Jest tests passed! This PR is ready to merge.

@stutijain2006
Copy link
Contributor Author

I don't think we need the AI integration here. But an interactive tutorial is not a terrible idea.

Heyyy @walterbender I have removed the commits which were integrating the AI model into the same and now only interactive menu is integrating with proper functioning and hence, would ask you to review it once again. These commits will now integrate the menu which will help the user to understand the things on better way.

@walterbender
Copy link
Member

Still a bit flaky. I get stuck at Step 4.
But do like the general idea.

Maybe we can sit down with Devin and design a script.

@stutijain2006
Copy link
Contributor Author

Still a bit flaky. I get stuck at Step 4. But do like the general idea.

Maybe we can sit down with Devin and design a script.

It works perfectly for me, do let me know when you and @pikurasa are up for a meet, we can decide the things there.

@walterbender
Copy link
Member

I am testing on Firefox running locally on Fedora.
Also, FWIW, in Step 3, after I open the palette and continue to Step 4, the palette closes before I can grab the block. I have to reopen it.

@stutijain2006
Copy link
Contributor Author

stutijain2006 commented Feb 6, 2026

I am testing on Firefox running locally on Fedora. Also, FWIW, in Step 3, after I open the palette and continue to Step 4, the palette closes before I can grab the block. I have to reopen it.

Yes It was made in that manner only, that on step-4 you have to open the palette and then drag the block into the space. If you wish then I can change it to the state where the palette will remain open and the person has to drag the block only.

Even all the steps which are related, will follow this pattern only.

@walterbender
Copy link
Member

That would make sense. But we really should discuss the script with Devin. And also figure out how to launch it. Maybe as an option from the ? button?

@stutijain2006
Copy link
Contributor Author

That would make sense. But we really should discuss the script with Devin. And also figure out how to launch it. Maybe as an option from the ? button?

Yes we should discuss the script, and then on the basis of that we can proceed, also we can launch it in 2-3 ways-

  1. We can launch it every time when user reloads the page but that seems to be somewhat less feasible and irritating.
  2. We can access it through a button on the toolbar.
  3. Or we can replace the current menu with it and it will appear in the same manner.

@walterbender
Copy link
Member

I think it would make sense to (1) have a card in the help (? button) sequence that lets you launch the tutorial and (2) maybe have a submenu for the ? button to choose between the cards and the tutorial. @pikurasa what do you think?

@github-actions
Copy link
Contributor

github-actions bot commented Mar 5, 2026

✅ All Jest tests passed! This PR is ready to merge.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 5, 2026

❌ Some Jest tests failed. Please check the logs and fix the issues before merging.

Failed Tests:

help.test.js

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants